Scan path visualization and comparison using visual aggregation techniques

We demonstrate the use of different visual aggregation techniques to obtain non-cluttered visual representations of scanpaths. First, fixation points are clustered using the mean-shift algorithm. Second, saccades are aggregated using the Attribute-Driven Edge Bundling (ADEB) algorithm that handles a saccades direction, onset timestamp, magnitude or their combination for the edge compatibility criterion. Flow direction maps, computed during bundling, can be visualized separately (vertical or horizontal components) or as a single image using the Oriented Line Integral Convolution (OLIC) algorithm. Furthermore, cosine similarity between two flow direction maps provides a similarity map to compare two scanpaths. Last, we provide examples of basic patterns, visual search task, and art perception. Used together, these techniques provide valuable insights about scanpath exploration and informative illustrations of the eye movement data

Study of pupil diameter and eye movements to enhance flight safety. Etude de diamètre pupillaire et de mouvements oculaires pour la sécurité aérienne

L'analyse d'événements aériens révèle que la plupart des accidents aéronautiques ont pour origine une surveillance inadaptée de paramètres de vol induite par une vigilance réduite, le stress ou une charge de travail importante. Une solution prometteuse pour améliorer la sécurité aérienne est d'étudier le regard des pilotes. La pupille est un bon indicateur de l'état attentionnel/cognitif tandis que les mouvements oculaires révèlent des stratégies de prises d'information. La question posée dans ce manuscrit est d'évaluer l'apport de l'oculométrie pour la sécurité aérienne par les contributions suivantes : 1-2) Les deux premières études de ce doctorat ont démontré que les effets d'interaction entre la luminance et la charge cognitive sur la réaction pupillaire. La composante pupillaire impactée dépend de la nature de la charge - soutenue ou transitoire. 3) Un cadre mathématique développé fournit un moyen d'illustration de schémas visuels pour l'analyse qualitative. Ce cadre ouvre également la voie à de nouvelles méthodes pour comparer quantitativement ces schémas visuels. 4) Une technique originale d'analyse de fixations et de construction d'un ratio "exploration-exploitation" est proposée et est appliquée dans deux cas d'études en simulateur de vol. 5) Enfin, on propose un cadre théorique d'intégration de l'oculométrie dans les cockpits. Ce cadre comporte quatre étapes présentées dans, à la fois, l'ordre chronologique de l'intégration et la complexité technique de réalisation.Most aviation accidents include failures in monitoring or decision-making which are hampered by arousal, stress or high workload. One promising avenue to further enhance the flight safety is looking into the pilots' eyes. The pupil is a good indicator of cognitive/attentional states while eye movements reveal monitoring strategies. This thesis reflected upon the application of eye tracking in aviation with following contributions: 1-2) The two pupil experiments revealed that the luminance impacts the cognitive pupil reaction. Depending on the nature of the cognitive load - sustained or transient - the corresponding pupillary component would be impacted. The same amount of cognitive load under dimmer luminance condition would elicit larger tonic pupil diameter in a sustained load paradigm and larger phasic pupil response in a transient load paradigm. 3) We designed a novel mathematical framework and method that provide comprehensive illustrations of scanpaths for qualitative analysis. This framework also makes a lane for new methods of scanpaths comparison. 4) The developed technique of analysis of fixations and construction of "explore-exploit" ratio is presented and verifed on the data from two experiments in flight simulators. 5) Eventually, we proposed a framework of eye tracking integration into the cockpits. It contains four stages presented in both chronological order of its integration and technical complexity

Uncertainty visualization of gaze estimation to support operator-controlled calibration

      In this paper, we investigate how visualization assets can support the qualitative evaluation of gaze estimation uncertainty. Although eye tracking data are commonly available, little has been done to visually investigate the uncertainty of recorded gaze information. This paper tries to fill this gap by using innovative uncertainty computation and visualization. Given a gaze processing pipeline, we estimate the location of this gaze position in the world camera. To do so we developed our own gaze data processing which give us access to every stage of the data transformation and thus the uncertainty computation. To validate our gaze estimation pipeline, we designed an experiment with 12 participants and showed that the correction methods we proposed reduced the Mean Angular Error by about 1.32 cm, aggregating all 12 participants’ results. The Mean Angular Error is 0.25° (SD=0.15°) after correction of the estimated gaze. Next, to support the qualitative assessment of this data, we provide a map which codes the actual uncertainty in the user point of view.

Intelligent cockpit: eye tracking integration to enhance the pilot-aircraft interaction

In this research, we use eye tracking to monitor the attentional behavior of pilots in the cockpit. We built a cockpit monitoring database that serves as a reference for real-time assessment of the pilot’s monitoring strategies, based on numerous flight simulator sessions with eye-tracking recordings. Eye tracking may also be employed as a passive input for assistive system, future studies will also explore the possibility to adapt the notifications’ modality using gaze

The impact of luminance on tonic and phasic pupillary responses to sustained cognitive load

Pupillary reactions independent of light conditions have been linked to cognition for a long time. However, the light conditions can impact the cognitive pupillary reaction. Previous studies underlined the impact of luminance on pupillary reaction, but it is still unclear how luminance modulates the sustained and transient components of pupillary reaction – tonic pupil diameter and phasic pupil response. In the present study, we investigated the impact of the luminance on these two components under sustained cognitive load. Fourteen participants performed a novel working memory task combining mathematical computations with a classic n-back task. We studied both tonic pupil diameter and phasic pupil response under low (1-back) and high (2-back) working memory load and two luminance levels (gray and white). We found that the impact of working memory load on the tonic pupil diameter was modulated by the level of luminance, the increase in tonic pupil diameter with the load being larger under lower luminance. In contrast, the smaller phasic pupil response found under high load remained unaffected by luminance. These results showed that luminance impacts the cognitive pupillary reaction – tonic pupil diameter (phasic pupil response) being modulated under sustained (respectively, transient) cognitive load. These findings also support the relationship between the locus-coeruleus system, presumably functioning in two firing modes – tonic and phasic – and the pupil diameter. We suggest that the tonic pupil diameter tracks the tonic activity of the locus-coeruleus while phasic pupil response reflects its phasic activity. Besides, the designed novel cognitive paradigm allows the simultaneous manipulation of sustained and transient components of the cognitive load and is useful for dissociating the effects on the tonic pupil diameter and phasic pupil response

Electro-Encephalography and Electro-Oculography in Aeronautics: A Review Over the Last Decade (2010–2020)

Electro-encephalography (EEG) and electro-oculography (EOG) are methods of electrophysiological monitoring that have potentially fruitful applications in neuroscience, clinical exploration, the aeronautical industry, and other sectors. These methods are often the most straightforward way of evaluating brain oscillations and eye movements, as they use standard laboratory or mobile techniques. This review describes the potential of EEG and EOG systems and the application of these methods in aeronautics. For example, EEG and EOG signals can be used to design brain-computer interfaces (BCI) and to interpret brain activity, such as monitoring the mental state of a pilot in determining their workload. The main objectives of this review are to, (i) offer an in-depth review of literature on the basics of EEG and EOG and their application in aeronautics; (ii) to explore the methodology and trends of research in combined EEG-EOG studies over the last decade; and (iii) to provide methodological guidelines for beginners and experts when applying these methods in environments outside the laboratory, with a particular focus on human factors and aeronautics. The study used databases from scientific, clinical, and neural engineering fields. The review first introduces the characteristics and the application of both EEG and EOG in aeronautics, undertaking a large review of relevant literature, from early to more recent studies. We then built a novel taxonomy model that includes 150 combined EEG-EOG papers published in peer-reviewed scientific journals and conferences from January 2010 to March 2020. Several data elements were reviewed for each study (e.g., pre-processing, extracted features and performance metrics), which were then examined to uncover trends in aeronautics and summarize interesting methods from this important body of literature. Finally, the review considers the advantages and limitations of these methods as well as future challenges

Integrating electrodes to headsets for human-system interaction and psycho-physiological monitoring

Research in human monitoring led to development of powerful tools for users-systems communication via recording electrophysiological data and sending them to the computer system. Eye and gaze tracking are standing as important methods for a broad range of human monitoring applications such as in neuroscience, psychology, industrial engineering, aeronautics, military, and medical expertise. Nowadays, the most used eye-tracking technique is video-based tracking based on infrared illumination. However, the tools using this technique present a certain number of disadvantages. Notably, for the head-mounted tools, such systems obstruct the visual field and therefore are not suitable for integration in real operational environments. An alternative technique is the electro-oculography (EOG) which consists in measuring the standing potential between the front and back of the eyes. This potential increases when the cornea approaches an electrode and it decreases when the cornea moves in the opposite direction. EOG represents one of the easiest methods to estimate eye movements by using low-cost low-energy consumption devices without obstructing the visual field nor handling infrared light. This technique is particularly convenient for the head-mounted peripherals such as audio or virtual reality headsets. Typical features of signal processing, such as the mean of fixation duration, saccadic rates, and blinks are explored in various environments including air traffic control, pilots and co-pilots of civil and military aircraft. Interestingly, the EOG, as well as electroencephalography (EEG) features were correlated to drivers’ and pilots’ workload, drowsiness and fatigue

Eliciting Sustained Mental Effort Using the Toulouse N-Back Task: Prefrontal Cortex and Pupillary Responses

In safety-critical environments such as piloting or air-traffic control, the study of mental overload is crucial to further reduce accident rates. However, researchers face the complexity of inducing an important amount of mental effort in laboratory conditions. Therefore, we designed a novel paradigm, named “Toulouse N-back Task” (TNT), combining the classical n-back task with a mathematical processing to replicate the multidimensional sustained high mental workload (MW) existing in many complex occupations. Instead of memorizing and comparing unique items, as in classical n-back task, participants have to memorize and to compare the results of mathematics operations. Twenty partic-ipants were tested with the TNT under three load factors (n = 0, 1, or 2) with functional Near-InfraRed Spectroscopy (fNIRS) and pupillary measurements. The results revealed that higher difficulty degraded the cognitive performance together with increased prefrontal oxygenation and an increase in pupil diameter. Hence, hemodynamic responses and pupil diameter were sensitive to different levels of TNT’s difficulty. This paradigm could serve as a viable alternative to the classical n-back task and enable the progressive increase of the difficulty, for example, to test “high performer” individuals

Évaluation de l’expérience de pilotage d’un avion léger en réalité virtuelle

En aéronautique, les cockpits en réalité virtuelle (RV) commencent à émerger, et il est désormais possible d’imaginer une application pour la formation professionnelle des pilotes. Il est néanmoins nécessaire d’évaluer cette technologie puisqu’elle peut entrainer une surcharge mentale pouvant être délétère sur l’apprentissage. Dans la présente étude, nous évaluons le degré d’immersion, de fatigue et de qualité d’interaction d’un simulateur en RV. Cinq pilotes instructeurs ont réalisé un scénario de vol à bord d’un Robin DR-400 à l’aérodrome de Lasbordes (LFCL, Toulouse) en environnement virtuel. L’évaluation de ce simulateur est réalisée à partir du recueil de données subjectives et de questionnaires. Les premiers résultats montrent que l’environnement RV est évalué très positivement (immersion, conduite du vol; pas de fatigue ni mal des transports rapportés). Toutefois, des limites apparaissent dans certaines phases critiques de vol, et la résolution du casque ne facilite pas la lecture de certains paramètres de vol

Hear me Flying! Does Visual Impairment Improve Auditory Display Usability during a Simulated Flight?

Sonification refers to systems that convey information into the non-speech audio modality [1], . This technique has been largely invested in developing guidance systems for visually impaired individuals. In 2008, more than 140 systems of this type used in various application areas were referenced [2], . In aeronautics, such a system –namely the Sound Flyer– is currently used by visually impaired pilots in real flight context to control the aircraft attitude. However, it is unclear if this system would be acceptable for sighted individuals. Indeed, early visual deprivation leads to compensatory mechanisms which often result in better auditory attentional skills [7,8]. In the present study we assessed this issue. Two groups of pilots (blind vs. sighted) took part in a flight simulator experiment. They were all blindfolded to avoid potential visual information acquisition (i.e. some blind individuals had residual visual capacities). Participants had to perform successive aircraft maneuvers on the sole basis of auditory information provided by the sound flyer. Maneuvers difficulty varied with the number of parameters to apply: easy (none), medium (one: pitch or bank) or hard (two: pitch and bank). The Sound Flyer generated a pure tone (53 dB SPL) modulated as a function of pitch (tonal variation) and bank (inter-aural and rhythmic variations). We assessed flight performance along with subjective (NASA-TLX) and neurological (irrelevant auditory-probe technique; [9], ) measures of cognitive workload. We hypothesized that the automatic cerebral reaction to deviant auditory stimuli (10% “ti” among 90% “ta”; 56db SPL) would be affected by the difficulty [10,11] and participants’ auditory attention. Preliminary data analyses revealed that blind and sighted participants reached target-attitudes with good accuracy (mean error of 2.04°). Globally, subjective cognitive workload and brain responses to the auditory probe were influenced by the difficulty of the maneuver but not by the visual impairment. These initial results provide evidence that auditory displays are effective, not only for maintaining straight and level flight [6], , but also for attaining precise aircraft attitudes. Results also suggest that flight maneuvers should remain quite simple to avoid too high cognitive workload. In other words, attitude sonification can provide robust information and, along with Brungart and Simpson [3] specifications, could contribute to the fight against spatial disorientation in the cockpit